Picture distribution system and method, picture distribution apparatus and a method therefor, picture receiving apparatus and a method therefore, and recording medium and program used therewith

ABSTRACT

A picture transmitting apparatus transmits, to a picture receiving apparatus, a picture as a predetermined area of an omnidirectional picture. The picture receiving apparatus displays a portion of the transmitted picture which corresponds to a viewpoint. When the viewpoint is moved, information of the movement is transmitted from the picture receiving apparatus to the picture transmitting apparatus, and a picture corresponding to the new viewpoint is transmitted from the picture transmitting apparatus to the picture receiving apparatus. The picture receiving apparatus displays a portion of the received picture which corresponds to the viewpoint.

BACKGROUND OF THE INVENTION

The present invention relates to picture distribution systems andmethods, picture distributing apparatuses and methods therefor, picturereceiving apparatuses and methods therefor, and recording media andprograms used therewith, and in particular, to a picture distributionsystem and method, a picture distribution apparatus and a methodtherefor, and a picture receiving apparatus and a method therefore inwhich pictures can be distributed without overloading a network, and arecording medium and a program which are used therewith.

A new picture-space service is in the process of being launched. In thenew picture-space service, from pictures having various viewpoints, suchas omnidirectional pictures, a user can selectively view a picture froma user-preferred viewpoint. Multi-viewpoint pictures such asomnidirectional pictures have large amounts of data since they includepictures constituting the space, other than a picture (or a displayedpicture) being actually viewed by the user.

When the streaming distribution of the multi-viewpoint pictures isperformed by a transmission link such as the Internet, in the existingstreaming distribution systems, such as REALSERVER® of RealVideo andQUICKTIME® of Apple, all pictures constituting the picture space aredistributed from a server to a client. This is because an applicationsuch as transmission of omnidirectional pictures is not assumed.

Japanese Unexamined Patent Application Publication No. 11-205772describes the network distribution of omnidirectional pictures. In theinvention in this Publication, the transmitting side transmits allpictures, while the receiving side separates a picture to be displayed.In this type of distribution method, there is a large load on networkbands since all the pictures are distributed despite that most of thepictures are not displayed. Also, when the displayed pictures havecompressed form, decoding size increases, thus increasing the processingload on the client. This causes a problem in that real-time processingis difficult.

Accordingly, an approach is required in which only the necessarypictures are distributed without distributing all the pictures. InJapanese Unexamined Patent Application Publication No. 2000-132674, asystem is proposed in which a viewpoint picture of omnidirectionalpictures is transmitted and received. In the proposed system, a clientuses coordinates to designate a viewpoint picture, and a servertransmits only a picture at the designated coordinates. The proposedsystem is based on the assumption that it is mainly used for stillpictures. Thus, in response to a user's urgent viewpoint-moving request,a picture from the requested viewpoint cannot be provided due to a delayin response on the network, so that a smooth viewpoint movement is notrealized.

SUMMARY OF THE INVENTION

The present invention is made in view of the above circumstances, and anobject thereof is to distribute pictures in which a viewpoint can besmoothly moved, without overloading a network and a receiving side.

According to an aspect of the present invention, a picture distributionsystem is provided which includes a transmitting apparatus, and areceiving apparatus. The transmitting apparatus distributes anomnidirectional picture to the receiving apparatus by a network. Thetransmitting apparatus includes a first receiving unit for receivingviewpoint information from the receiving apparatus, an extracting unitfor, based on the received viewpoint information, extracting apredetermined area as a first picture from the omnidirectional picture,and a first transmitting unit for transmitting, to the receivingapparatus, unit regions obtained by dividing the first picture. Thereceiving apparatus includes an acquiring unit for acquiring theviewpoint information, a second transmitting unit for transmitting theacquired viewpoint information to the transmitting apparatus, a secondreceiving unit for receiving the first picture transmitted from thesecond transmitting apparatus, a storage unit for storing the firstpicture received by the second receiving unit, and an output unit foroutputting a second picture as a predetermined area extracted from thestored first picture, the second picture corresponding to the acquiredviewpoint information.

According to another aspect of the present invention, a picturedistribution method for distributing an omnidirectional picture from atransmitting apparatus to a receiving apparatus by a network isprovided. The transmitting apparatus performs a first receiving step forreceiving viewpoint information from the receiving apparatus, anextracting step for, based on the received viewpoint information,extracting a predetermined area as a first picture from theomnidirectional picture, and a first transmitting step for transmitting,to the receiving apparatus, unit regions obtained by dividing the firstpicture. The receiving apparatus performs an acquiring step foracquiring the viewpoint information, a second transmitting step fortransmitting the acquired viewpoint information to the transmittingapparatus, a second receiving step for receiving the first picturetransmitted from the transmitting apparatus, a storage step for storingthe first picture received in the second receiving step, and an outputstep for outputting a second picture as a predetermined area extractedfrom the stored first picture, the second picture corresponding to theacquired viewpoint information.

According to another aspect of the present invention, a picturedistribution apparatus for distributing an omnidirectional picture to areceiving apparatus by a network is provided. The picture distributingapparatus includes a receiving unit for receiving viewpoint informationfrom the receiving apparatus, an extracting unit for, based on thereceived viewpoint information, extracting a predetermined area as apicture from the omnidirectional picture, and a transmitting unit fortransmitting, to the receiving apparatus, predetermined unit regionsobtained by dividing the picture. Preferably, based on a motion vectorincluded in the received viewpoint and a round trip time of the network,the extracting unit determines the next predetermined area to beextracted from the omnidirectional picture.

The picture distribution apparatus may further include a calculatingunit for calculating the margin of a predetermined area which is outputby the receiving apparatus in response to the viewpoint information, andwhich corresponds to the predetermined area transmitted to the receivingapparatus by the transmitting unit. Based on the received viewpointinformation and the calculated margin, the extracting unit may extractthe predetermined area as the picture from the omnidirectional picture.

According to another aspect of the present invention, a picturedistribution method for a picture distribution apparatus fordistributing an omnidirectional picture to a receiving apparatus by anetwork is provided. The picture distribution method includes the stepsof receiving viewpoint information from the receiving apparatus,extracting, based on the received viewpoint information, a predeterminedarea as a picture from the omnidirectional picture, and transmitting, tothe receiving apparatus, predetermined unit regions obtained by dividingthe picture.

According to another aspect of the present invention, a first recordingmedium containing a computer-readable program for a picture distributionapparatus for distributing an omnidirectional picture to a receivingapparatus is provided. The program includes the steps of receivingviewpoint information from the receiving apparatus, extracting, based onthe received viewpoint information, a predetermined area as a picturefrom the omnidirectional picture, and transmitting, to the receivingapparatus, predetermined unit regions obtained by dividing the picture.

According to another aspect of the present invention, a first programfor a computer controlling a picture distribution apparatus fordistributing an omnidirectional picture by a network is provided. Theprogram causes the computer to execute the steps of receiving viewpointinformation from the receiving apparatus, extracting, based on thereceived viewpoint information, a predetermined area as a picture fromthe omnidirectional picture, and transmitting, to the receivingapparatus, predetermined unit regions obtained by dividing the picture.

According to another aspect of the present invention, a picturereceiving apparatus for receiving an omnidirectional picture distributedfrom a transmitting apparatus by a network is provided. The picturereceiving apparatus includes an acquiring unit for acquiring viewpointinformation, a transmitting unit for transmitting the acquired viewpointinformation to the transmitting apparatus, a receiving unit forreceiving a first picture as a predetermined area extracted from theomnidirectional picture, the first picture being transmitted from thetransmitting apparatus, a storage unit for storing the-received firstpicture, and an output unit for outputting a second picture as apredetermined area extracted from the stored first picture, the secondpicture corresponding to the acquired viewpoint information.

The picture receiving apparatus further includes a calculating unit forcalculating the margin of the second picture which corresponds to thefirst picture. Based on the calculated margin, the transmitting unit maytransmit the viewpoint information to the transmitting apparatus.

According to another aspect of the present invention, a picturereceiving method for a picture receiving apparatus for receiving anomnidirectional picture distributed from a transmitting apparatus by anetwork is provided. The picture receiving method includes the steps ofacquiring viewpoint information, transmitting the acquired viewpointinformation to the transmitting apparatus, receiving a first picture asa predetermined area extracted from the omnidirectional picture, thefirst picture being transmitted from the transmitting apparatus, storingthe received first picture, and outputting a second picture as apredetermined area extracted from the stored first picture, the secondpicture corresponding to the acquired viewpoint information.

According to another aspect of the present invention, a second recordingmedium containing a computer-readable program for a picture receivingapparatus for receiving an omnidirectional picture distributed from atransmitting apparatus by a network is provided. The program includesthe steps of acquiring viewpoint information, transmitting the acquiredviewpoint information to the transmitting apparatus, receiving a firstpicture as a predetermined area extracted from the omnidirectionalpicture, the first picture being transmitted from the transmittingapparatus, storing the received first picture, and outputting a secondpicture as a predetermined area extracted from the stored first picture,the second picture corresponding to the acquired viewpoint information.

According to another aspect of the present invention, a second programfor a computer controlling a picture receiving apparatus for receivingan omnidirectional picture distributed from a transmitting apparatus bya network is provided. The program causes the computer to execute thesteps of acquiring viewpoint information, transmitting the acquiredviewpoint information to the transmitting apparatus, receiving a firstpicture as a predetermined area extracted from the omnidirectionalpicture, the first picture being transmitted from the transmittingapparatus, storing the received first picture, and outputting a secondpicture as a predetermined area extracted from the stored first picture,the second picture corresponding to the acquired viewpoint information.

According to a picture distribution system and method of the presentinvention, viewpoint information is transmitted from a receivingapparatus to a transmitting apparatus, and based on the viewpointinformation, a predetermined area is extracted as a first picture froman omnidirectional picture. The first picture is divided into unitregions and is transmitted from the transmitting apparatus to thereceiving apparatus.

According to a picture distribution apparatus and method, and arecording medium and a program which are used therewith, based onviewpoint information received from a receiving apparatus, apredetermined area is extracted as a picture from an omnidirectionalpicture. The picture is divided into predetermined unit regions, and theregions are transmitted to the receiving apparatus.

According to a picture receiving apparatus and method, and a recordingmedium and a program which are used therewith, viewpoint information istransmitted to a transmitting apparatus, and based on the viewpointinformation, a first picture transmitted from the transmitting apparatusis stored, and a predetermined area is extracted as the stored firstpicture and is output.

Additional features and advantages of the present invention aredescribed in, and will be apparent from, the following DetailedDescription of the Invention and the Figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an omnidirectional picturedistribution system to which the present invention is applied;

FIG. 2 is an illustration of an operation of the omnidirectional picturedistribution system shown in FIG. 1 when a viewpoint is changed;

FIG. 3 is an illustration of principles of viewpoint movement control;

FIG. 4 is an illustration of RTT measurement processing based on RTCP;

FIG. 5 is an illustration of picture division into tiles;

FIG. 6 is an illustration of state changes of RTSP;

FIG. 7 is an illustration of state changes of RTSP when the methodMOVEPLAY is added;

FIG. 8 is a flowchart illustrating a process performed in the INIT stateof a picture transmitting apparatus 1;

FIG. 9 is a flowchart illustrating a process performed in the READYstate of the picture transmitting apparatus 1;

FIG. 10 is a flowchart illustrating a process performed in the PLAYstate of the picture transmitting apparatus 1;

FIG. 11 is a flowchart illustrating a process performed in the MOVEPLAYstate of the picture transmitting apparatus 1;

FIG. 12 is an illustration of state changes in a picture receivingapparatus 2;

FIG. 13 is a flowchart illustrating a process performed in the INITstate of the picture receiving apparatus 2;

FIG. 14 is a flowchart illustrating a process performed in the READYstate of the picture receiving apparatus 2;

FIG. 15 is a flowchart illustrating a process performed in the PLAYstate of the picture receiving apparatus 2;

FIG. 16 is a flowchart illustrating a process performed in the MOVEPLAYstate of the picture receiving apparatus 2; and

FIG. 17 is a flowchart illustrating a process performed in the MOVEstate of the picture receiving apparatus 2.

DETAILED DESCRIPTION OF THE INVENTION

In the present invention, a distribution method is employed in which,when omnidirectional pictures are distributed through a network, bytransmitting a picture corresponding to a viewpoint selected by a user,and the periphery, a response can be made to an urgent viewpoint-movingrequest from the user. The present invention is specifically describedbelow with reference to the accompanying drawings.

At first, the configuration and operation of the entire system of thepresent invention are described with reference to FIGS. 1 to 4. Afterthat, more detailed points are explained with reference to FIGS. 5 to17.

FIG. 1 is a block diagram showing an omnidirectional picturedistribution system to which the present invention is applied. Theomnidirectional picture distribution system includes a picturetransmitting apparatus 1 which generates a multi-viewpoint picturestream such as omnidirectional pictures, and at least a picturereceiving apparatus 2 which receives the picture stream from the picturetransmitting apparatus 1. The picture transmitting apparatus 1 and thepicture receiving apparatus 2 are connected to each other by a network 3including the Internet, a local area network (LAN), a wide area network(WAN), etc.

The picture transmitting apparatus 1 includes a picture input unit 11, astream control unit 12, a viewpoint-information receiving unit 13, and astream transmitting unit 14. A plurality of picture receivingapparatuses 2 are connected to the picture transmitting apparatus 1 bythe network 3.

The picture input unit 11 includes, for example, a videocamera, and adisk device storing picture files. A picture input from the pictureinput unit 11 is transferred to the stream control unit 12. The picturetransferred to the stream control unit 12 may be a compression-codedpicture stream.

The viewpoint-information receiving unit 13 receives viewpointinformation from the picture receiving apparatus 2, and includes anetwork interface. The viewpoint information from theviewpoint-information receiving unit 13 and the picture stream from thepicture input unit 11 are input to the stream control unit 12, and itgenerates a stream suitable for a viewpoint picture for the picturereceiving apparatus 2.

The picture stream output from the stream control unit 12 may be aplurality of streams generated by spatially diving a picture into aplurality of blocks. A method for the division into blocks is, forexample, tile division in JPEG (Joint Photographic Experts Group) 2000.The details are described later with reference to FIG. 5.

The stream transmitting unit 14 uses the Real-time Transport Protocol(RTP) or the like to convert the picture stream into packets, andtransmits the packets to the picture receiving apparatus 2. The streamtransmitting unit 14 also includes a network interface.

The picture receiving apparatus 2 includes a viewpoint-information inputunit 21, a display-picture control unit 22, a viewpoint-informationtransmitting unit 23, a stream receiving unit 24, and a picture outputunit 25. The viewpoint-information input unit 21 has, for example, inputdevices such as a mouse, a keyboard, a head tracker, and a joystick.Viewpoint information input from the viewpoint-information input unit 21is transferred to the display-picture control unit 22 and theviewpoint-information transmitting unit 23.

The viewpoint-information transmitting unit 23 transmits the viewpointinformation to the picture transmitting apparatus 1, and includes anetwork interface. The stream receiving unit 24 includes a networkinterface transfers a received picture stream to the display-picturecontrol unit 22. Based on the viewpoint information, the display-picturecontrol unit 22 generates a picture to be displayed from the receivedpicture stream. The display-picture control unit 22 also performssynchronizing processing for the case of receiving a plurality ofpicture streams. The picture output unit 25 also includes, for example,a display device or the like, and outputs and displays the picturegenerated by the display-picture control unit 22.

FIG. 2 shows the communication between the picture receiving apparatus 2and the picture transmitting apparatus 1 when, in the system of FIG. 1,the viewpoint in the picture receiving apparatus 2 changes. Based on theviewpoint information received from the picture receiving apparatus 2through the network 3 and the viewpoint-information receiving unit 13,the stream control unit 12 in the picture transmitting apparatus 1extracts, from the picture input unit 11, a transmitting picture 202which is a predetermined area (part) of the entire spatial picture 201among multi-viewpoint pictures such as omnidirectional pictures. Insteps S1, S2, etc., the stream transmitting unit 14 transmits theextracted picture 202 to the picture receiving apparatus 2 through thenetwork 3.

In the picture receiving apparatus 2, the display-picture control unit22 temporarily stores in a built-in storage portion the picture 202received through the stream receiving unit 24. The display-picturecontrol unit 22 captures, from the viewpoint-information input unit 21,information (viewpoint information) relating to a viewpoint(coordinates) based on a user's operation, and re-generates, from thestored picture 202, a display picture in the area corresponding to theviewpoint. The display-picture control unit 22 outputs and displays there-generated picture as a user's-viewpoint picture 204 on the pictureoutput unit 25.

When, in the picture receiving apparatus 2, occurrence of a viewpointmovement in the direction indicated by the arrow 205 is input to theviewpoint-information input unit 21 based on a user's operation, in stepS11, the viewpoint-information transmitting unit 23 notifies the picturetransmitting apparatus 1 of the viewpoint movement by using the network3. If the viewpoint-information transmitting unit 23 immediatelynotifies the picture transmitting apparatus 1 of the viewpoint movementafter the viewpoint movement occurs, a certain time (hereinafterreferred to as a “response time TD”) is required until the picturereceiving apparatus 2 receives a picture reflecting the viewpointmovement, due to a reason such as a delay in transmission on the network3 and a delay in processing by the picture transmitting apparatus 1.During a period (the response time TD) until the picture reflecting theviewpoint change reaches the picture receiving apparatus 2, thedisplay-picture control unit 22 determines a picture 207 in a newdisplay position matching the direction of the movement, and displaysthe picture 207 on the picture output unit 25.

In step S3, in the picture transmitting apparatus 1, based on theviewpoint-movement information received through theviewpoint-information receiving unit 13, the stream control unit 12immediately determines a picture 208 in the position of the next pictureto be transmitted, inputs the picture 208 from the picture input unit11, and transmits the picture 208 from the stream transmitting unit 14.

When receiving the picture 208, which reflects the viewpoint movement,through the stream receiving unit 24, the display-picture control unit22 in the picture receiving apparatus 2 stores the received picture 208,and determines and extracts, from the stored picture 208, a picture 210in the position corresponding to the coordinates (viewpoint) input fromthe viewpoint-information input unit 21. The display-picture controlunit 22 outputs and displays the extracted picture 208 on the pictureoutput unit 25. The above operations are continuously executed by thepicture transmitting apparatus 1 and the picture receiving apparatus 2.

FIG. 3 shows viewpoint movement control using the relationship betweenreceived (transmitting) picture size and displayed picture size. Asshown in FIG. 2, after the picture receiving apparatus 2 transmits aviewpoint movement notification, the viewpoint movement cannot beimmediately reflected in a received picture. Thus, within the responsetime TD, control must be performed so that a displayed picture 302 isnot positioned out of the frame of a received picture 301. Thedisplay-picture control unit 22 employs, as this control, any one of thefollowing methods.

In a first method, the upper limit of a viewpoint moving speed in thepicture receiving apparatus 2 is set based on displayed picture margins(the vertical margin 303 and the horizontal margin 303 in FIG. 3) andthe response time TD. The vertical margin 303 represents the distancebetween a horizontal end (the upper end in the example of FIG. 3) of thedisplayed picture 302 and a horizontal end (the upper end in the exampleof FIG. 3) of the received picture 301. The margin 304 represents thedistance between a vertical end (the right end in FIG. 3) of thedisplayed picture 302 and a vertical end (the right end in FIG. 3) ofthe received picture 301.

In this case, the upper limit of the viewpoint moving speed is definedby the following expression:(Upper Limit of Viewpoint Moving Speed)≦(Picture Margin)/(ResponseTime)  (1)

When the movement direction is arbitrary, the movement direction isdecomposed into a horizontal component and a vertical component forcalculation. For example, as shown in FIG. 3, when the viewpoint movesin the direction indicated by the arrow (motion vector) 305, themovement direction of the motion vector 305 is decomposed into ahorizontal component 307 and a vertical component 306 for calculation.When the angle between the movement direction and the horizontaldirection is represented by θ, the following expression holds:(Moving Speed in θ Direction)≦Min(Horizontal Component 304/cos θ,Vertical Component 303/sin θ)/Response Time  (2)where Min (A, B) represents the minimum between A and B, and thehorizontal component 304 and the vertical component 303 represent theabsolute values of lengths, respectively. Similarly, expression (2) canbe applied to viewpoint zooming-in or zooming-out by assuming that thedisplayed picture moves in four directions.

In a second method, the transmitting picture size is set based on theresponse time, with the viewpoint moving speed regarded as constant. Inthis case, transforming expression (1) gives the following expression:(Picture Margin)≧(Viewpoint Moving Speed)×(Response Time)  (3)Thus, the transmitting picture size is represented by the followingexpression:(Transmitting Picture Size)≧(Viewpoint Moving Speed)×(ResponseTime)+(Displayed Picture Size)  (4)

In a third method, the number of pixels is set based on the responsetime, with the transmission picture size and the viewpoint moving speedregarded as constant. By transforming expression (4), while payingattention to the following points:

-   displayed picture size=the number of displayed pixels; and    transmitting picture size=received picture size, the following    expression is obtained:    (Number of Displayed Pixels)≦(Received Picture Size)−(Viewpoint    Moving Speed)×(Response Time)  (5)    In this case, it is possible that the third method be combined with    processing such as enlargement by digital zooming of superficial    displayed-picture size for a reduction in the number of displayed    pixels.

In a fourth method, the above first to third methods are flexiblyswitched in proportional to the length of the response time. In the caseof a long response time, there is a possibility that the first methodmay cause a very delayed viewpoint movement. Accordingly, a method ofswitching to the second method is possible.

The response time used in the fourth method is shown as follows:Response Time=Buffering Time in Picture Receiving Apparatus 2+Round Trip(Transmission Delay) Time on Network (RTT)+Processing Time in PictureTransmitting Apparatus 1Each value is made known by exchanging monitoring information betweenthe picture transmitting apparatus 1 and the picture receiving apparatus2.

Also, the RTT included in the response time can be measured by thefollowing manner. Measurement of an RTT using Real-time Control Protocol(RTCP) uses a sender report (SR) which is periodically transmitted fromthe picture transmitting apparatus 1 to the picture receiving apparatus2 and a receiver report (RR) with which the picture receiving apparatus2 responds to the picture transmitting apparatus 1. Each of the SR andthe RR is one of messages in RTCP. In the SR, its transmission time iswritten, and in the RR, the transmission time written in the SR and atime up to transmission of the RR after receiving the SR are written.

FIG. 4 shows that the picture transmitting apparatus 1 and the picturereceiving apparatus 2 exchange an SR and an RR. At time T1, the picturetransmitting apparatus 1 transmits an SR 401 to the picture receivingapparatus 2. Then, in the SR 401, the transmission time T1 is written.The display-picture control unit 22 in the picture receiving apparatus 2receives the SR 401 at time T2, calculates a one-way delay time (RTT/2)by using the following expression:One-way Delay Time (RTT/2)=T2−T1  (6)and generates an RTT by doubling the one-way delay time.

This processing is based on the assumption that the internal clocks ofthe picture transmitting apparatus 1 and the picture receiving apparatus2 synchronize with each other. Although a method for establishing thesynchronization between the internal clocks is not described, thesynchronization can be established by using Network Time Protocol (NTP)which is commonly used in the Internet, or an original protocol for NTP.

After a predetermined time (“delay”) elapses, at time T3, the picturereceiving apparatus 2 transmits the RR 402. Then, in the RR 402, thetransmission time T1 and a time (delay=T3−T2) from reception of the SR401 up to transmission of the RR 402 are written. The stream controlunit 12 in the picture transmitting apparatus 1 receives the RR 402 atthe time T4, and calculates an RTT by using the following expression:RTT=T4−T1−delay=T4−T1−(T3−T2)  (7)In the calculation of the RTT by the picture transmitting apparatus 1,the synchronization between the internals clocks of the picturetransmitting apparatus 1 and the picture receiving apparatus 2 does notalways need to be established because the RR includes the “delay” time.The method in expression (7) is also described in RFC (Request forcomments) 1889.

As described above, by using the present invention, when multi-viewpointpictures, such as omnidirectional pictures, are distributed by thenetwork 3, bands used on the network 3 and the processing load in areceiving terminal can be reduced, so that picture-spatial service on abasis of omnidirectional pictures can be provided even to relativelynarrow band networks such as wireless networks and powerless terminalssuch as personal digital assistants (PDAs) and cellular phones.

In a case in which the pictures are based on Motion JPEG (JointPhotographic Experts Group) 2000, when a partial picture is distributed,it is not necessary to separately distribute information indicating thatthe distributed picture corresponds to which of the entire picturebecause JPEG 2000 has a tile division function as an encoding functionfor dividing a screen into partial screens, so that the protocol betweenthe transmitter and the receiver can be simplified, thus enablingfacilitated control of viewpoint movement.

Next, a specific example for realizing the above-described system isdescribed. At first, the data format of omnidirectional pictures aredescribed. The picture input unit 11 converts the data format ofomnidirectional pictures into a cylindrical picture format. The pictureinput unit 11 uses JPEG 2000 as a picture compressing method. The use ofJPEG 2000 produces the following advantages.

JPEG 2000 is suitable for transmission of omnidirectional picturesbecause it does not need the difference between image frames. As inomnidirectional pictures, when picture size constituting the entirespace is very large, a partial picture in an arbitrary viewpointdirection is only shown, and the viewpoint is moved at random by theuser, prediction encoding used in MPEG or the like is almost useless.Also, when considering transmission on the Internet, MPEG which needsinterframe difference has a problem in that, when a packet loss occurs,an error is transmitted to the subsequent frames. This problem does notoccur in JPEG 2000 since it does not need interframe difference.

In JPEG 2000, after a picture is divided into a plurality of arbitraryareas called “tiles”, only some tiles can be decoded. Each tile size canbe set to be constant in a size designated in an encoding mode, and thetile size is described in a header in JPEG 2000. In addition, indexnumbers are assigned to the tiles from the top left to bottom right ofthe entire picture. Accordingly, the numbering has an advantage in thatknowing a tile index number can easily specify which portion of theentire picture the tile picture is. Moreover, each portion can beseparately encoded. Thus, by controlling encoding of a picture so thatthe quality of portions (e.g., the upper end and lower end of a spatialpicture mapped in a cylindrical shape) to which the user does not pay alot of attention is reduced to increase a compression factor, morestorage area and network bandwidths can be saved.

As described with reference to FIG. 2, this picture distribution systemuses a method for distributing a viewpoint picture and a picture (e.g.,the image 202 in FIG. 2) which consists of an arbitrary number of tilesand which includes its periphery. Only the required number of tiles canbe distributed because JPEG 2000 has a function in which, even if atruncate image is input to a decoder (the display-picture control unit22), it can be decoded. FIG. 5 shows an example in which an image isdivided into ten tiles. The tiles are sequentially numbered 0 to 9 asindex numbers. Accordingly, for example, the coordinates of the top leftof the tile image of index 3 can be easily known as (tile_width×3, 0).As shown in FIG. 5, “tile_width” represents a tile width, and“tile-height” represents a tile height.

Next, a protocol for use in viewpoint movement is described. In aprotocol, used between the picture receiving apparatus 2 (client) andthe picture transmitting apparatus 1 (server) when the viewpoint in thepicture receiving apparatus 2 changes, the following points areconsidered. Although an example of Real Time Streaming Protocol (RTSP)in extended form is described here, a new original protocol may bedesigned. Since extension of RTSP generates portions in stream controlwhich can be used in common, it is only required to define simplecommands.

Conditions for consideration are as follows:

-   (1) A viewpoint-moving request must be transmitted by using a    reliable communication link or a communication protocol. This    corresponds to the TCP.-   (2) A viewpoint moving direction or a motion vector must be    transmitted from the client to the server. The motion vector    includes not only a direction of movement but also representation of    a movement (moving speed) per unit time.-   (3) It is preferable for the server to notify the client of    estimated arrival information of a partial picture reflecting the    viewpoint-moving request. The estimated arrival information is    information that the client should know beforehand and that    indicates from which packet the coordinates of a received picture    will change. In the case of using RTP to transmit pictures, the    estimated arrival information includes time-stamp information and    sequence numbers of RTP packets. By allowing the client to know the    estimated arrival information beforehand, the upper limit of the    viewpoint moving speed on the receiving side can be easily    determined.-   (4) It is preferable that a partial picture to be transmitted be    easily re-formed.-   (5) It is preferable that the coordinates of a received partial    picture be easily found.

Next, a specific example of the protocol is described. The extended RTSPis described below. The details of RTSP are disclosed in RFC 2326. RTSPcontrols delivery of real-time data in the application level, and canuse TCP as a transport-layer protocol. This satisfies the abovecondition (1).

RTSP, PLAY, STOP, PAUSE, etc., are defined as stream-control commands,but a watching and listening method such as viewpoint movement is notassumed. Accordingly, new commands corresponding to viewpoint movementare defined. RTSP has a lot of extensibility, and new commands can beeasily defined. Command adding techniques can include a technique foradding new parameters to conventionally existing methods (PLAY, STOP,PAUSE, etc.), and a technique for adding a new method. Between these,the technique for adding a new method is preferable because a new stateis generated by adding a command. Accordingly, an example of the methodaddition is described below. However, the technique for adding newparameters has an advantage in that mounting is simplified, comparedwith the method addition.

RTSP has state transition as shown in FIG. 6. By adding a viewpointmoving command, the state transition is changed to that shown in FIG. 7.In FIGS. 6 and 7, the encircled portions indicate states, and the arrowsindicate changes. An explanation of each arrow is a method name, and thearrow indicates that activation of the method causes a state change.

By way of example, in the RTSP state transition shown in FIG. 6, byactivating the method TEARDOWN in the PLAY state, the state is changedinto the INIT state. Also, by activating the PAUSE state in the PLAYstate, the state is changed into the READY state. The INIT state ischanged into the READY state by activating the method SETUP. The READYstate is changed into the INIT state by activating the method TEARDOWN,and is changed into the PLAY state by activating the method PLAY.

In the RTSP state transition (shown in FIG. 7) to which the methodMOVEPLAY is added, in addition to the state changes shown in FIG. 6,state changes occur among the MOVEPLAY state, the INIT state, the PLAYstate, and the READY state.

Specifically, the MOVEPLAY state is changed into the INIT state byactivating the method TEARDOWN, is changed into the PLAY state byactivating the method MOVEPLAY, and is changed into the READY state byactivating the method PAUSE. The PLAY state is changed into the MOVEPLAYstate by activating the method MOVEPLAY.

In the present invention, the method MOVEPLAY is added. MOVEPLAY is amethod designating viewpoint movement, and is sent at the start ofviewpoint movement. Parameters interpreted by the method MOVEPLAY areidentical to those of the method PLAY, but differ from those of themethod PLAY in vector parameters representing viewpoint movement andzoom parameters representing zooming. When the motion vector of theviewpoint or zooming factor does not change, and when viewpoint movementis continuously performed, it is not necessary to sent the methodMOVEPLAY to the server again.

Next, the vector parameter is described. Occurrence of the viewpointmovement causes the client to send, to the server, the followingrequest:

-   -   MOVEPLAY rtsp://video.example.com/video RTSP/1.0    -   Cseq: 835    -   Session: 123456    -   Vector:xy 10-30

The first viewpoint-movement stopping request designates a moving speedof zero.

The common format of the vector parameter is as follows:

-   -   Vector:xy (Moving Speed in X-axis Direction) [(Moving Speed in        Y-axis Direction)] or    -   Vector:polar (Rotational Angle with X-axis set to 0°) [(Moving        Speed)]

For example, when the X-Y coordinate system is used for designation suchas “vector:xy 10-30”, the designation indicates that viewpoint movementis performed at a speed of 10 pixels in the X direction (the rightdirection) and −30 pixels (i.e., downwardly 30 pixels) in the Ydirection (the upward direction) per unit time (one second) in the X-Ycoordinate system. When the moving speed in the Y direction is zero, therepresentation for the Y axis may be omitted.

Also, for example, when the polar coordinate system is used to designatea motion vector, the designation indicates that viewpoint movement isperformed at a speed of 20 pixels in an anticlockwise 90-degreedirection (upward) per unit time (one second). When the viewpoint movingspeed is fixed, designation is easier in the polar coordinate systemthan in the X-Y coordinate system. When the moving speed is fixed, andthe agreement between the server and the client is obtained as anapplication specification or by negotiation at the start of a session,the moving speed may be omitted.

Next, the zoom parameter is described. The zoom parameter is used torequest zooming in or zooming out, and is treated similarly to thevector parameter. The zoom parameter is shown as follows:

-   -   Zoom:(Zoom Speed)

In this zoom parameter, a positive value equal to zero or greater is setas the zoom speed. The value 1.0 represents 1× magnification zooming, avalue greater than 1.0 represents zooming in (enlarging direction), anda value less than 1.0 represents zooming out (reducing direction). Anenlargement factor or a reduction factor is greater as it is furtherthan 1.0. For example, in the case of a zooming speed of 2.0, the speedrepresents zooming a double magnification per frame, and in the case ofa zooming speed of 3.0, the speed represents zooming at a triple speedper frame. The server responds with an error when a zooming speed equalto zero or less is designated, or further zooming is impossible.

Next, the method MOVESTOP is described. The method MOVESTOP is a commanddesignating a viewpoint-movement stop and is used to stop viewpointmovement. Parameters interpreted by the method MOVESTOP are similar tothose interpreted by the method STOP.

Next, replies of the server are described. When the server receives themethod MOVEPLAY or MOVESTOP, and cannot interpret the received method,it replies with “Method Not Allowed” as defined in RFC 2326. When theserver can interpret the method, it replies in accordance with themethod PLAY or STOP. In order to indicate from which sequence therequest is reflected, the replay include the following RTP-Info header:

-   -   RTSP/1.0 200 OK    -   Cseq:835    -   Session:123456    -   RTP-Info:url=rtsp://video.examplecompany/video;seq=3456;rtptime=789012

In order to fulfill the roll of transmitting a picture in response to arequest from the picture receiving apparatus 2, the picture transmittingapparatus 1 has state transition similar to that in RTSP. Accordingly,the picture transmitting apparatus 1 has four states, the INIT state,the READY state, the PLAY state, and the MOVEPLAY state. Processescorresponding to the four states are shown in FIGS. 8 to 11. Step S12(FIG. 8) in the INIT state, steps S51, S52, and S55 (FIG. 10) in thePLAY state, and the entire process (FIG. 11) of the MOVEPLAY state addedsteps added by extending the RTSP method.

As shown in FIG. 8, after the process is started, the stream controlunit 12 sets the INIT state in step S11, and executes initialization oftransmitting tile numbers in step S12. In step S13, the stream controlunit 12 determines whether the method SETUP has been received. If it hasnot been received yet, in step S14, the stream control unit 12determines whether a termination signal has been received. If thetermination signal has not been received, the stream control unit 12returns to step S13, and repeats the subsequent steps. In step S13, ifit is determined that the method SETUP has been received, the streamcontrol unit 12 proceeds to step S15, and the state is changed into theREADY state (FIG. 7). In step S14, if it is determined that thetermination signal has been received, the process ends.

In the READY state shown in FIG. 9, in step S31, the stream control unit12 determines whether the method PLAY has been received. If it has notbeen received yet, the stream control unit 12 proceeds to step S32, anddetermines whether the method TEARDOWN has been received. If it isdetermined that the method TEARDOWN has not been received, the streamcontrol unit 12 returns to step S31, and repeats the subsequent steps.

In step S31, if it is determined that the method PLAY has been received,the stream control unit 12 proceeds to step S33, and changes the stateinto the PLAY state (FIG. 7). In step S32, it is determined that themethod TEARDOWN has been received, the stream control unit 12 proceedsto step S34, and changes the state into the INIT state (FIG. 7).

In the PLAY state shown in FIG. 10, in step S51, the stream control unit12 transmits a selected tile. In step S52, the stream control unit 12determines whether the method MOVEPLAY has been received. If the methodMOVEPLAY has been received, the stream control unit 12 proceeds to stepS55, and changes the state into the MOVEPLAY state (FIG. 7). In stepS52, if it is determined that the method MOVEPLAY has not been received,the stream control unit 12 proceeds to step S53, and determines whetherthe method PAUSE has been received. If it is determined that the methodPAUSE has been received, the stream control unit 12 proceeds to stepS56, and changes the state into the READY state (FIG. 7).

In step S53, if it is determined that the method PAUSE has not beenreceived, the stream control unit 12 proceeds to step S54, anddetermines whether the method TEARDOWN has been received. If the methodTEARDOWN has been received, the stream control unit 12 proceeds to stepS57, and changes the state into the INIT state (FIG. 7). In step S54, ifit is determined that the method TEARDOWN has not been received, thestream control unit 12 returns to step S51, and repeats the subsequentsteps.

In the MOVEPLAY state shown in FIG. 11, in step S71, the stream controlunit 12 sets RTT in the elapsed time. The RTT is a measured valuedescribed with reference to FIG. 4. In step S72, the stream control unit12 calculates a motion based on the following expression:Motion=Motion Vector×Elapsed TimeThe value set in step S71 is used as the elapsed time.

In step S73, the stream control unit 12 executes the process forselecting a transmitting tile again. In step S74, the stream controlunit 12 controls the stream transmitting unit 14 to transmit thetransmitting tile selected in step S73. In step S75, the stream controlunit 12 determines whether the method MOVESTOP has been received. If ithas been received, in step S79, the stream control unit 12 changes thestate into the PLAY state (FIG. 7).

In step S75, if the method MOVESTOP has not been received, the streamcontrol unit 12 proceeds to step S76, and determines whether the methodPAUSE has been received. If it is determined that the method PAUSE hasbeen received, the stream control unit 12 proceeds to step S80, andexecutes transmission of all the tiles. In step S81, the stream controlunit 12 changes the state into the READY state (FIG. 7).

In step S76, if the method PAUSE has been received, the stream controlunit 12 proceeds to step S77, and determines whether the TEARDOWN hasbeen received. If it is determined that the method TEARDOWN has beenreceived, the stream control unit 12 proceeds to step S80 and changesthe state into the INIT state (FIG. 7).

In step S77, if it is determined that the method TEARDOWN has beenreceived, the stream control unit 12 proceeds to step S78 and sets thereciprocal of the frame rate in the elapsed time. After that, the streamcontrol unit 12 returns to step S72, and repeats the subsequent steps.

The picture receiving apparatus 2 must respond to an interactive requestfrom the user other than the protocol operations with the picturetransmitting apparatus 1. Accordingly, in the picture receivingapparatus 2, the MOVE state, which is a closed state in the picturereceiving apparatus 2, is added to the states in FIG. 7 of thetransmitting side, so that the picture receiving apparatus 2 has statetransition shown in FIG. 12. Each state in FIG. 12 is identical inmeaning to that in FIG. 7. A state change is performed in response to auser's request. The arrows of the state changes are labeled user'srequests.

In the example in FIG. 12, when a STOP request is made in the PLAYstate, the PLAY state is changed into the INIT state. When a PAUSErequest is made, the PLAY state is changed into the READY state. When aviewpoint-moving request is made, the PLAY state is changed into theMOVEPLAY state.

When a PLAY request is made in the INIT state, the INIT state is changedinto the READY state. When a PLAY request is made in the READY state,the READY state is changed into the PLAY state. When a STOP request ismade, the READY state is changed into the INIT state. When aviewpoint-moving request is made, the READY state is changed into theREADY state.

When a viewpoint stopping request is made in the MOVE state, the MOVEstate is changed into the READY state. When a STOP request is made inthe MOVEPLAY state, the MOVEPLAY state is changed into the INIT state.When a viewpoint stopping request is made, the MOVEPLAY state is changedinto the PLAY state. When a PAUSE request is made, the MOVEPLAY state ischanged into the READY state.

FIGS. 13 to 17 show processes for the states. Steps S123, S125, S129,and S130 (FIG. 14) in the READY state, steps S142, S143, S146, and S147(FIG. 15) in the PLAY state, and the entire processes of the MOVEPLAYstate (FIG. 16) and the MOVE state (FIG. 17) are added processes. In theMOVE state, no messages are exchanged with the picture transmittingapparatus 1.

Referring to the INIT state in FIG. 13, in step S101, the INIT state isset. In step S102, the display-picture control unit 22 determineswhether a PLAY request has been made. If it is determined that the PLAYrequest has been made, the process proceeds to step S104, and thedisplay-picture control unit 22 executes transmission of the methodSETUP. In step S105, the display-picture control unit 22 changes thestate into the READY state (FIG. 12).

In step S102, if it is determined that the PLAY request has not beenmade, the process proceeds to step S103, and the display-picture controlunit 22 determines whether a termination signal has been generated. Ifit is determined that the termination signal has not been generated, theprocess returns to step S102, and the subsequent steps are repeatedlyexecuted. If it is determined that the termination signal has beengenerated, the process is terminated.

In the READY state in FIG. 14, the display-picture control unit 22determines in step S121 whether a PLAY request has been made. If it hasnot been made, in step S122, the display-picture control unit 22determines whether a STOP request has been made. If it has not beenmade, the display-picture control unit 22 proceeds to step S123 anddetermines whether a viewpoint-moving request has been made. If it isdetermined that the viewpoint-moving request has also not been made, thesubsequent steps are repeatedly executed.

In step S121, if it is determined that the PLAY request has been made,the display-picture control unit 22 proceeds to step S124 and transmitsthe method PLAY. In step S125, the picture receiving apparatus 2executes initialization of a margin value of a received picturecorresponding to a viewpoint picture. After that, the display-picturecontrol unit 22 proceeds to step S126 and changes the state into thePLAY state (FIG. 12).

In step S122, if it is determined that the STOP request has been made,the display-picture control unit 22 proceeds to step S127 and transmitsthe method TEARDOWN. In step S128, the display-picture control unit 22changes the state into the INIT state (FIG. 12).

In step S123, if it is determined the viewpoint-moving request has beenmade, the display-picture control unit 22 proceeds to step S129 anddetermines whether all the tiles have been received. If they have notbeen received, the display-picture control unit 22 returns to step S121and continuously executes the subsequent steps. In step S129, if it isdetermined that all the tiles have been received, the display-picturecontrol unit 22 proceeds to step S130 and changes the state into theMOVE state (FIG. 12).

In the PLAY state in FIG. 15, in step S141, the display-picture controlunit 22 receives a tile picture. In step S142, the display-picturecontrol unit 22 separates a portion corresponding to the viewpointpicture from the tile picture received and stored in the built-in memoryin step S141, and outputs and displays the separated portion on thepicture output unit 25.

In step S143, the display-picture control unit 22 determines whether theviewpoint-moving request has been made. If it has not been made, thedisplay-picture control unit 22 proceeds to step S144 and determineswhether the PAUSE request has been made. When determining that even thePAUSE request has not been made, the display-picture control unit 22proceeds to step S145 and determines whether the STOP request has beenmade. If it has not been made, the display-picture control unit 22returns to step S141, and repeatedly executes the subsequent steps.

In step S143, if it is determined that the viewpoint-moving request hasbeen made, the display-picture control unit 22 proceeds to step S146 andexecutes transmission of the method MOVEPLAY. In step S147, thedisplay-picture control unit 22 changes the state into the MOVEPLAYstate (FIG. 12).

In step S144, if it is determined that the PAUSE request has been made,the display-picture control unit 22 proceeds to step S148 and transmitsthe method PAUSE. The display-picture control unit 22 executes, in stepS149, reception of all the tiles through the stream receiving unit 24,and changes the state into the READY state in step S150 (FIG. 12).

In step S145, if it is determined that the STOP request has been made,the display-picture control unit 22 proceeds to step S151, and transmitsthe method TEARDOWN. In step S150, the display-picture control unit 22changes the state into the INIT state (FIG. 12).

In the MOVEPLAY state in FIG. 16, when the display-picture control unit22 receives the picture in step S161, it determines, in step S162,whether, in the picture received in step S161, the viewpoint-movingrequest is reflected. If it is not reflected, the display-picturecontrol unit 22 proceeds to step S163 and updates a margin value in themoving direction. In other words, this increases or reduces the marginin the moving direction by the motion.

In step S162, if it is determined that in the received picture theviewpoint-moving request is reflected, the display-picture control unit22 proceeds to step S164 and executes re-calculation of the marginvalue. After steps S163 and S164, the display-picture control unit 22proceeds to step S165. In step S165, the display-picture control unit 22extracts a picture corresponding to the viewpoint input from theviewpoint-information input unit 21, and outputs and displays theextracted picture on the picture output unit 25.

Proceeding to step S166, the display-picture control unit 22 determineswhether the motion vector has been changed. If it has not been changed,the display-picture control unit 22 returns to step S161 andcontinuously executes the subsequent steps.

In step S166, if it is determined that the motion vector has beenchanged, the display-picture control unit 22 proceeds to step S167, andexecutes re-calculation of the motion vector. In other words, thedisplay-picture control unit 22 further acquires, from theviewpoint-information input unit 21, the moving direction, andcalculates the moving speed based on the following expression:Moving Speed=Margin/RTT

Next, proceeding to step S168, the display-picture control unit 22transmits the method MOVEPLAY. In step S169, the display-picture controlunit 22 determines whether the viewpoint stopping request has been made.If it has not been made, the display-picture control unit 22 proceeds tostep S170 and determines whether the PAUSE request has been made. Wheneven the PAUSE request has not been made, the display-picture controlunit 22 proceeds to step S171 and determines whether the STOP requesthas been made. If it has not been made, the display-picture control unit22 returns to step S161 and continuously executes the subsequent steps.

In step S169, if it is determined that the viewpoint stopping requesthas been made, the display-picture control unit 22 proceeds to step S172and transmits the method MOVESTOP. In step S173, the display-picturecontrol unit 22 changes the state into the PLAY state (FIG. 12).

In step S170, if it is determined that the PAUSE request has been made,the display-picture control unit 22 proceeds to step S174 and transmitsthe method PAUSE. In step S175, the display-picture control unit 22executes reception of all the tiles through the stream receiving unit24. In step S176, the display-picture control unit 22 changes the stateinto the READY state (FIG. 12).

In step S171, if it is determined that the STOP request has been made,the display-picture control unit 22 proceeds to step S177 and executestransmission of the method TEARDOWN. In step S178, the display-picturecontrol unit 22 changes the state into the INIT state (FIG. 12).

In the process of the MOVE state, in step S191, the display-picturecontrol unit 22 executes calculation of the motion based on thefollowing expression:Motion=Motion Vector/Frame Rate

Next, proceeding to step S192, the display-picture control unit 22separates a picture corresponding to the viewpoint information from theviewpoint-information input unit 21, and outputs and displays thepicture on the picture output unit 25. In step S193, the display-picturecontrol unit 22 determines whether the viewpoint has been moved. If ithas been moved, the display-picture control unit 22 returns to step S191and continuously executes the subsequent steps (FIG. 12). In step S193,if it is determined that the viewpoint has not been move, thedisplay-picture control unit 22 proceeds to step S194 and changes thestate into the READY state.

An idea that only a portion being actually viewed is transmitted can beapplied also to the case of constituting a car navigation system usingthe Internet. In this case, by storing map information of the entiretyin the form of pictures of JPEG 2000, and performing division into tilesin units of, for example, degrees of latitude and longitude, a controlmethod for the system can be handled similarly to distribution ofpartial omnidirectional pictures.

As described above, the picture receiving apparatus 2 calculates themargin of a portion which is actually output (displayed) by the picturereceiving apparatus 2 and which corresponds to a portion of a picturetransmitted from the picture transmitting apparatus 1 to the picturereceiving apparatus 2. However, the calculation may be executed by thepicture transmitting apparatus 1. In this case, based on the viewpointinformation from the picture receiving apparatus 2, and the calculatedmargin, the picture transmitting apparatus 1 determines and extractsfrom the omnidirectional picture a portion to be transmitted to thepicture receiving apparatus 2, and transmits the portion to the picturereceiving apparatus 2.

At this time, the picture transmitting apparatus 1 can determine thesize of the picture for transmission by measuring a response time withRTCP. Also, the picture receiving apparatus 2 can determine and notifythe picture transmitting apparatus 1 of the size of the picture.

In addition, at this time, before initiating a session, in a set-upmode, or during a session, for example, by using RTSP, both the picturetransmitting apparatus 1 and the picture receiving apparatus 2 mustmutually obtain agreement about the moving speed of the viewpoint.

The above consecutive processing can be executed by hardware, but may beexecuted by software. When software is used to execute the consecutiveprocessing, programs constituting the software are installed from anetwork or a recording medium into a computer built into dedicatedhardware or into, for example, a multi-purpose personal computer inwhich by installing various programs, various functions can be executed.

The types of the recording medium, not only includes magnetic disks(including floppy disks), optical disks (including CD-ROMs and DVDs),magneto-optical disks (MiniDiscs), and package media including asemiconductor memory, but also include ROMs and hard disks containingprograms.

In this Specification, steps constituting a program recorded on therecording medium include, not only processes which are time-sequentiallyperformed in accordance with described order, but also processes whichare executed in parallel or separately if they are not always performedtime-sequential order. In this Specification the system represents theentirety of an apparatus including a plurality of units.

As described above, according to the present invention, a picture can bedistributed from a transmitting apparatus to a receiving apparatus.Also, even if the viewpoint is moved, the receiving apparatus can outputand display, in real time, a picture corresponding to the new viewpoint.

It should be understood that various changes and modifications to thepresently preferred embodiments described herein will be apparent tothose skilled in the art. Such changes and modifications can be madewithout departing from the spirit and scope of the present invention andwithout diminishing its intended advantages. It is therefore intendedthat such changes and modifications be covered by the appended claims.

1. A picture distribution system comprising: a receiving apparatus; anda transmitting apparatus in communication with said receiving apparatusfor distributing omnidirectional pictures to said receiving apparatusthrough a network; said transmitting apparatus including, firstreceiving means for receiving viewpoint information from said receivingapparatus; extracting means for, based on the received viewpointinformation, extracting a predetermined area as a first picture from anomnidirectional picture to be sent to said receiving apparatus; andfirst transmitting means for transmitting, to said receiving apparatus,unit regions obtained by dividing the first picture; and said receivingapparatus including: acquiring means for acquiring the viewpointinformation; second transmitting means for transmitting the acquiredviewpoint information to said transmitting apparatus; second receivingmeans for receiving the first picture transmitted from said secondtransmitting apparatus; storage means for storing the first picturereceived by said second receiving means; and output means for outputtinga second picture as a predetermined area extracted from the stored firstpicture, the second picture corresponding to the acquired viewpointinformation, wherein, based on a motion vector included in the receivedviewpoint information and a transmission delay of said network, saidextracting means determines the next predetermined area to be extractedfrom the omnidirectional picture.
 2. A picture distribution method fordistributing an omnidirectional picture from a transmitting apparatus toa receiving apparatus by a network, comprising the steps of: a firstreceiving step for receiving viewpoint information from said receivingapparatus; an extracting step for, based on the received viewpointinformation, extracting a predetermined area as a first picture from anomnidirectional picture to be distributed to said receiving apparatus; afirst transmitting step for transmitting, to said receiving apparatus,unit regions obtained by dividing the first picture; an acquiring stepfor acquiring the viewpoint information transmitted to said transmittingapparatus; a second transmitting step for transmitting the acquiredviewpoint information to said transmitting apparatus; a second receivingstep for receiving the first picture transmitted from said transmittingapparatus; a storage step for storing the first picture received in saidsecond receiving step; an output step for outputting a second picture asa predetermined area extracted from the stored first picture, the secondpicture corresponding to the acquired viewpoint information; and adetermining step for determining the next predetermined area to beextracted from the omnidirectional picture based on a motion vectorincluded in the received viewpoint information and a transmission delayof said network.
 3. A picture distribution apparatus for distributingomnidirectional pictures to a receiving apparatus through a network,comprising: receiving means for receiving viewpoint information fromsaid receiving apparatus; extracting means for, based on the receivedviewpoint information, extracting a predetermined area as a picture froman omnidirectional picture to be distributed to the receiving apparatus;transmitting means for transmitting, to said receiving apparatus,predetermined unit regions obtained by dividing the picture; anddetermining means for determining the next predetermined area to beextracted from the omnidirectional picture based on a motion vectorincluded in the received viewpoint information and a transmission delayof said network.
 4. A picture distribution apparatus according to claim3, further comprising calculating means for calculating the margin ofthe predetermined area which is output by said receiving apparatus inresponse to the viewpoint information, and which corresponds to thepredetermined area transmitted to said receiving apparatus by saidtransmitting means, wherein, based on the received viewpoint informationand the calculated margin, said extracting means extracts thepredetermined area as the picture from the omnidirectional picture.
 5. Apicture distribution method for a picture distribution apparatus fordistributing an omnidirectional picture to a receiving apparatus by anetwork, comprising the steps of: receiving viewpoint information fromsaid receiving apparatus; extracting, based on the received viewpointinformation, a predetermined area as a picture from the omnidirectionalpicture to be distributed to the receiving apparatus; transmitting, tosaid receiving apparatus, predetermined unit regions obtained bydividing the picture; and determining the next predetermined area to beextracted from the omnidirectional picture based on a motion vectorincluded in the received viewpoint information and a transmission delayof said network.
 6. A recording medium containing a computer-readableprogram for a picture distribution apparatus for distributing anomnidirectional picture to a receiving apparatus, the program causingthe apparatus to execute comprising the steps of: receiving viewpointinformation from said receiving apparatus; extracting, based on thereceived viewpoint information, a predetermined area as a picture froman omnidirectional picture to be distributed to the receiving apparatus;transmitting, to said receiving apparatus, predetermined unit regionsobtained by dividing the picture; and determining the next predeterminedarea to be extracted from the omnidirectional picture based on a motionvector included in the received viewpoint information and a transmissiondelay of a network.
 7. A program stored in a computer readable mediumfor a computer controlling a picture distribution apparatus fordistributing an omnidirectional picture by a network, said programcausing the computer to execute the steps of: receiving viewpointinformation from said receiving apparatus; extracting, based on thereceived viewpoint information, a predetermined area as a picture fromthe omnidirectional picture to be distributed; transmitting, to saidreceiving apparatus, predetermined unit regions obtained by dividing thepicture; and determining the next predetermined area to be extractedfrom the omnidirectional picture based on a motion vector included inthe received viewpoint information and a transmission delay of saidnetwork.
 8. A picture receiving apparatus for receiving anomnidirectional picture distributed from a transmitting apparatusthrough a network, said picture receiving apparatus comprising:acquiring means for acquiring viewpoint information; transmitting meansfor transmitting the acquired viewpoint information to said transmittingapparatus; receiving means for receiving a first picture as apredetermined area extracted from the omnidirectional picture, the firstpicture being transmitted from said transmitting apparatus; storagemeans for storing the received first picture; output means foroutputting a second picture as a predetermined area extracted from thestored first picture, the second picture corresponding to the acquiredviewpoint information; and determining means for determining the nextpredetermined area to be extracted from the omnidirectional picturebased on a motion vector included in the acquired viewpoint informationand a transmission delay of said network.
 9. A picture receivingapparatus according to claim 8, further comprising calculating means forcalculating the margin of the second picture which corresponds to thefirst picture, wherein, based on the calculated margin, saidtransmitting means transmits the viewpoint information to saidtransmitting apparatus.
 10. A picture receiving method for a picturereceiving apparatus receiving an omnidirectional picture distributedfrom a transmitting apparatus through a network, said picture receivingmethod comprising the steps of: acquiring viewpoint information;transmitting the acquired viewpoint information to said transmittingapparatus; receiving a first picture as a predetermined area extractedfrom the omnidirectional picture, the first picture being transmittedfrom said transmitting apparatus; storing the received first picture;outputting a second picture as a predetermined area extracted from thestored first picture, the second picture corresponding to the acquiredviewpoint information; and determining the next predetermined area to beextracted from the omnidirectional picture based on a motion vectorincluded in the acquired viewpoint information and a transmission delayof said network.
 11. A recording medium containing a computer-readableprogram for a picture receiving apparatus for receiving anomnidirectional picture distributed from a transmitting apparatus by anetwork, the program causing the apparatus to execute comprising thesteps of: acquiring viewpoint information; transmitting the acquiredviewpoint information to said transmitting apparatus; receiving a firstpicture as a predetermined area extracted from the omnidirectionalpicture, the first picture being transmitted from said transmittingapparatus; storing the received first picture; and outputting a secondpicture as a predetermined area extracted from the stored first picture,the second picture corresponding to the acquired viewpoint information;determining the next predetermined area to be extracted from theomnidirectional picture based on a motion vector included in theacquired viewpoint information and a transmission delay of said network.12. A program stored in a computer readable medium for a computercontrolling a picture receiving apparatus for receiving anomnidirectional picture distributed from a transmitting apparatus by anetwork, said program causing the computer to execute the steps of:acquiring viewpoint information; transmitting the acquired viewpointinformation to said transmitting apparatus; receiving a first picture asa predetermined area extracted from the omnidirectional picture, thefirst picture being transmitted from said transmitting apparatus;storing the received first picture; outputting a second picture as apredetermined area extracted from the stored first picture, the secondpicture corresponding to the acquired viewpoint information; anddetermining the next predetermined area to be extracted from theomnidirectional picture based on a motion vector included in theacquired viewpoint information and a transmission delay of said network.